Pinging @elastic/ml-core (Team:ML)

Would changing the test so that field_2 is 10 times field_1 instead of 2 times also help with making it more consistent, since it would make the difference between using the correct value and an incorrect one easier to spot?

The way we determine the prediction error also seems like it could be improved, since using the absolute value of the difference between the predicted and actual value is very forgiving for small numbers and very strict for large numbers (being allowed to be off by 3 when the predicted value is 2 is a huge margin of error compared to being allowed to be off by 3 when the predicted value is 600).

Perhaps instead of the absolute difference between predicted and actual, the relative difference could be used, since that would allow the final assertion to be that the predictions are on average within some % of the actual value, rather than being within some absolute range. Something like this perhaps (I chose 1% arbitrarily, I don't know how lenient the test actually needs to be):

int expectedValue = 2 * featureValue;
predictionErrorPercentSum += Math.abs(predictionValue - expectedValue) / expectedValue;
...
double meanPredictionErrorPercent = predictionErrorPercentSum / sourceData.getHits().getHits().length;
// Assert that the predicted values are on average within 1% of the expected values
assertThat(meanPredictionErrorPercent, lessThanOrEqualTo(0.01));

Would changing the test so that field_2 is 10 times field_1 instead of 2 times also help with making it more consistent, since it would make the difference between using the correct value and an incorrect one easier to spot?

The way we determine the prediction error also seems like it could be improved, since using the absolute value of the difference between the predicted and actual value is very forgiving for small numbers and very strict for large numbers (being allowed to be off by 3 when the predicted value is 2 is a huge margin of error compared to being allowed to be off by 3 when the predicted value is 600).

Perhaps instead of the absolute difference between predicted and actual, the relative difference could be used, since that would allow the final assertion to be that the predictions are on average within some % of the actual value, rather than being within some absolute range. Something like this perhaps (I chose 1% arbitrarily, I don't know how lenient the test actually needs to be):

int expectedValue = 2 * featureValue;
predictionErrorPercentSum += Math.abs(predictionValue - expectedValue) / expectedValue;
...
double meanPredictionErrorPercent = predictionErrorPercentSum / sourceData.getHits().getHits().length;
// Assert that the predicted values are on average within 1% of the expected values
assertThat(meanPredictionErrorPercent, lessThanOrEqualTo(0.01));

@DonalEvans , thank you for your comment. This is a useful technique, and we employ similar assertions elsewhere in the test, where we actually verify that the algorithm is learning and predicting correct values. The goal of this test is to test that the alias fields are used correctly. Without fixing the hyperparameters on this test, the prediction was (on very rare occasions) completely off, and hence it would fail whether the assertion threshold is expressed in terms of relative or absolute values.

@DonalEvans , thank you for your comment. This is a useful technique, and we employ similar assertions elsewhere in the test, where we actually verify that the algorithm is learning and predicting correct values. The goal of this test is to test that the alias fields are used correctly. Without fixing the hyperparameters on this test, the prediction was (on very rare occasions) completely off, and hence it would fail whether the assertion threshold is expressed in terms of relative or absolute values.

Thanks for the explanation!

💚 Backport successful

Status	Branch	Result
✅	8.19
✅	9.1
✅	8.18
✅	9.0